Concerning the problem of barrier among passengers and unstable illumination on the bus, a detection and tracking algorithm was proposed based on edge feature and local invariant feature of head-shoulder. Firstly, the algorithm used adaptive threshold background subtraction method to achieve passenger segmentation. Secondly, it used Histogram of Oriented Gradient (HOG) feature of different sample sets to train Support Vector Machine (SVM) classifiers, and combined Adaptive Boosting (AdaBoost) algorithm to extract a strong classifier. And then it scanned the foreground using strong classifier to achieve passenger detection. Lastly, it extracted Speeded-Up Robust Feature (SURF) of target region and current search region, and then matched feature points to achieve passenger tracking. The experimental results show that this algorithm has detection rate and tracking rate of more than 80% in the case of barrier among passengers and unstable illumination, and it can meet the requirement of real-time. It can be used for passenger flow counting.